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DETAILED ACTION 

1 . An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 

1 .3 12. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

Authorization for this examiner's amendment was given in the telephone interview with 
Attorney Mr. Thomas C. Basso on February 6, 2006. In this communication, claims 1, 22, 23, 
26-28, and 30-35 have been amended, and claims 14 and 29 are cancelled. 

2. The claims have been amended as follows: 

Claim 1 (currently amended): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 

an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; and 
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one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; 

said environment type computer being adapted to execute; 

a processing operation of recognizing the identification information and the position 
information of each of said physical objects by identifying the visual marker on the surface of 
each of said physical objects; 

a processing operation of recognizing 4be-a_digital object dropped to a site on the surface 
of each of said physical objects; and 

a processing operation of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects. 

Claim 2 (original): The information input/output system according to claim 1, wherein at 
least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers. 

Claim 3 (original): The information input/output system according to claim 1, wherein at 
least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers; and 
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said environment type computer can expand a mouse/cursor operation on the installed 
portable computer onto said operation surfaces. 

Claim 4 (original): The information input/output system according to claim 1, wherein 

said imaging means can identify the position indicated by an optical pointer for indicating 
a specific position by irradiating a beam of light with a predetermined wavelength; and 

the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer. 

Claim 5 (original): The information input/output system according to claim 1, wherein 
said environment type computer executes a processing operation of calling the linked digital 
object and/or displaying the digital object to the user in response to a user operation applied to 
the site of forming the link information on the surface of each of the physical objects on the basis 
of the image picked up by said imaging means. 

Claim 6 (previously presented): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 
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one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 

an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; and 

a portable ID recognition device adapted to recognize/identify the visual marker on the 
surface of each of said physical objects. 

Claim 7 (previously presented): The information input/output system according to claim 
6, wherein 

said environment type computer is adapted to execute; 

a processing operation of receiving the identification information of the source object and 
that of the destination object from said ID recognition device; and 

a processing operation of applying an action specified on the basis of the combination of 
the type of the source object and that of the destination object. 
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Claim 8 (original): The information input/output system according to claim 6, wherein 

at least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers. 



Claim 9 (previously presented): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 

an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; and 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; 

at least one of said physical objects being a virtual camera having a virtual imaging 
direction as specified by the position information contained in the visual marker. 
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Claim 10 (previously presented): The information input/output system according to 
claim 9, wherein 

said environment type computer is adapted to execute; 

a processing operation of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction; 

a processing operation of generating a virtual picked up image according to the position 
information and the virtual imaging direction; and a processing operation of displaying the 
virtual picked up image to the user. 

Claim 1 1 (original): The information input/output system according to claim 9, wherein 
at least one of said physical objects is a portable computer that can move in said information 
space and exchange digital objects with another computer. 

Claim 12 (previously presented): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 
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an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; and 

an optical pointer adapted to point to a specific position by irradiating a beam of light 
with a predetermined wavelength, wherein said imaging means is capable of identifying the 
position pointed to by said optical pointer. 

Claim 13 (original): The information input/output system according to claim 12, wherein 
said environment type computer executes on the basis of the image picked up by said imaging 
means; 

a processing operation of recognizing the user operation on said operation surfaces, using 
said optical pointer; and 

a processing operation of controlling the display of the digital object by said display 
means according to the result of the recognition. 

Claim 14 (canceled). 
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Claim 15 (previously presented): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 

an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; 

one or more than one portable computers capable of exchanging digital objects with other 
computers; 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; and 

a portable ID recognition device adapted to recognize the visual marker on the surface of 
each of said physical objects and being capable of holding or releasing the recognized 
identification information. 

Claim 16 (original): The information input/output system according to claim 15, wherein 
said environment type computer executes; 
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a processing operation of obtaining a digital object corresponding to the held 
identification information in response to the holding operation of said ID recognition device; and 

a processing operation of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the releasing operation of 
said ID recognition device. 



Claim 17 (previously presented): An information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; 

a display means for displaying an image on said operation surfaces; 

an imaging means for picking up an image of said operation surfaces; 

a stationary environment type computer arranged in said information space; 

one or more than one portable computers capable of exchanging digital objects with other 
computers; and 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof, said visual marker including 
identification information and position information for said physical objects; 
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said environment type computer being adapted to allow mouse/cursor operations on said 
portable computers to extend to said operation surfaces and display a rubber band as visual 
feedback according to the area on said operation surfaces defined by means of said 
mouse/cursor. 

Claim 18 (original): The information input/output system according to claim 17, wherein 
said environment type computer is adapted to pick up an image of the area defined by the rubber 
band by means of said imaging means and take the picked up image in said information space as 
digital object. 

Claim 19 (original): The information input/output system according to claim 17, wherein 
another object can apply a process or a method it possesses to the rubber-banded original digital 
object in response to an action of said original object of being dropped on said another object. 

Claim 20 (original): The information input/output system according to claim 19, wherein 
said another object is a physical object possessing a process or a method for printing a document. 
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Claim 21 (original): The information input/output system according to claim 19, wherein 
said another object is a name card possessing a process or a method for mailing a document to 
the mail address corresponding to the name on it. 



Claim 22 (currently amended): An information input/output method. system to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method comprising: 

providing an information input/output system information --space having one or more than 
one operation surfaces arranged in said information space, a display means for displaying an 
image on said operation surfaces, an imaging means for picking up an image of said operation 
surfaces, a stationary environment type computer in said information space and one or more than 
one physical objects mounted on said operation surfaces and containing a visually identifiable 
visual marker on the surface thereof,. said. visual „m 
position.]^ said s ystem comprising :; 

a step of firstly recognizing the identification information and the position information of 
each of said physical objects from the visual marker included on the surface of each of said 
physical objects; 

a step of secondly recognizing the digital object dropped to a site on the surface of each 
of said physical objects; and 
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a step of forming link information for linking the digital object to the dropped site on the 
surface for each of said physical objects. 

Claim 23 (currently amended): The information input/output sy st em method. according 
to claim 22, wherein 

at least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers; and 

said environment type computer can expand a mouse/cursor operation on the installed 
portable computer onto said operation surfaces in said second recognition step. 

Claim 24 (currently amended): The information input/output systeffl"-method„ according 
to claim 22, wherein said imaging means can identify the position indicated by an optical pointer 
for indicating a specific position by irradiating a beam of light with a predetermined wavelength; 
and 

the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer. 
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Claim 25 (currently amended): The information input/output 5ystem-;m(gthpd..according 
to claim 22, further comprising: 

a step of calling the linked digital object and/or displaying the digital object to the user in 
response to a user operation applied to the site of forming the link information on the surface of 
each of the physical objects. 



Claim 26 (currently amended): An information input/output method.system to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method compri 

providing an information input/output system having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, a 
stationary environment type computer arranged in said information space, one or more than one 
physical objects mounted on said operation surfaces and containing a visually identifiable visual 
marker on the surface thereof, where the visual marker includes identification information and 
position information for the physical objects; and a portable ID recognition device adapted to 
recognize/identify the visual marker on the surface of each of said physical objects, ; said system 
comprising:^ 

a step of receiving the identification information of the source object and that of the 
destination object from said ID recognition device; and 
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a step of applying an action specified on the basis of the combination of the type of the 
source object and that of the destination object. 

Claim 27 (currently amended): An information input/output methpd_,$ystem-to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method comprising; 

providing an information input/output system having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, a 
stationary environment type computer in said information space and one or more than one 
physical objects mounted on said operation surfaces and containing a visually identifiable visual 
marker on the surface thereof, where the visual marker includes identification information and 
position information for the physical objects, at least one of said physical objects being a virtual 
camera having a virtual imaging direction as specified by the position information contained in 
the visual marker,- said system comprising:; 

a step of recognizing/identifying said virtual camera and identifying the position 
information and the virtual imaging direction; 

a step of generating a virtual picked up image according to the position information and 
the virtual imaging direction; and 

a step of displaying the virtual picked up image to the user. 
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Claim 28 (currently amended): An information input/output method„system"to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method comprising; 

providing an information input/output system having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, a 
stationary environment type computer arranged in said information space, one or more than one 
physical objects mounted on said operation surfaces and containing a visually identifiable visual 
marker on the surface thereof, where the visual marker includes identification information and 
position information for the physical objects, and an optical pointer adapted to point a specific 
position by irradiating a beam of light with a predetermined wavelength, said imaging means 
being capable of identifying the position pointed by said optical pointer, said -system 
comprising:; 

a step of recognizing the user operation on said operation surfaces, using said optical 
pointer; and 

a step of controlling the display of the digital object by said display means according to 
the result of the recognition. 



Claim 29 (canceled). 
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Claim 30 (currently amended): An information input/output method.system-to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method cpm^ 

providing an information input/output system having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, a 
stationary environment type computer arranged in said information space and one or more than 
one portable computers capable of exchanging digital objects with other computers, one or more 
than one physical objects mounted on said operation surfaces and containing a visually 
identifiable visual marker on the surface thereof, where the visual marker includes identification 
information and position information for the physical objects, and a portable ID recognition 
device adapted to recognize the visual marker on the surface of each of said physical objects and 
being capable of holding or releasing the recognized identification informationv - said -system 
comp r ising:^ 

a step of obtaining a digital object corresponding to the held identification information in 
response to the holding operation of said ID recognition device; and 

a step of transferring a digital object corresponding to the held identification information 
to a nearby physical object in response to the releasing operation of said ID recognition device. 
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Claim 31 (currently amended): An information input/output method system to be used 
for user operations relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said method com^ 

providing an information input/output system having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, a 
stationary environment type computer arranged in said information space, one or more than one 
portable computers capable of exchanging digital objects with other computers and one or more 
than one physical objects mounted on said operation surfaces and containing a visually 
identifiable visual marker on the surface thereof, where the visual marker includes identification 
information and position information for the physical objects- said- system -comprising:; 

a step of allowing mouse/cursor operations on said portable computers to extend to said 
operation surfaces; and 

a step of displaying a rubber band as visual feedback according to the scope on said 
operation surfaces as defined by means of said mouse/cursor. 

Claim 32 (currently amended): The information input/output method, system -according 
to claim 31, further comprising: 

a step of picking up an image of the scope defined by the rubber band by means of said 
imaging means and taking the picked up image in said information space as digital object. 
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Claim 33 (currently amended): The information input/output systemmethod. according 
to claim 31, further comprising: 

a step of applying a process or a method possessed by said another object to the rubber- 
banded original digital object in response to an action of said original object of being dropped on 
said another object. 

Claim 34 (currently amended): The information input/output syst e m method, according 
to claim 33, wherein said another object is a physical object possessing a process or a method for 
printing a document. 

Claim 35 (currently amended): The information input/output system- method.. according 
to claim 33, wherein said another object is name card possessing a process or a method for 
mailing a document to the mail address corresponding to the name on it. 

Allowable Subject Matter 

3. Claims 1-13, 15-28, and 30-35 are allowed. 

4. The following is an examiner's statement of reasons for allowance in combination with 
other claim limitations: 

Independent claims 1, 6, 9, 12, 15, 17, 22, 26, 27, 28, and 30-31, when considered 
as a whole, are allowable over the Prior Art of record. Specifically, the Prior Art of 
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record fails to teach that using the device/visual marker to pick up one or more physical 
objects in the information space, wherein the physical objects mounted on the operation 
surfaces and containing the visually identifiable visual marker on the surface thereof the 
visual marker including identification information and position information the physical 
objects, the processing operation of recognizing the digital object dropped to the site on 
the surface of each of the physical objects and forming link information for linking the 
digital object to the dropped site on the surface for each the physical objects. 
Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to True T. Chuong whose telephone number is 571-272-4134. The 
examiner can normally be reached on M-Th and alternate Fridays 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Weilun Lo can be reached on (571) 272-4847. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

True T. Chuong 
02/06/06 
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SUPERVISORY R4XENT EXAMINER 



